{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In this lab we'll demonstrate several common techniques and helpful tools used in a model building process:\n",
    "\n",
    "- Use Sklearn to generate polynomial features and rescale them\n",
    "- Create folds for cross-validation\n",
    "- Perform a grid search to optimize hyper-parameters using cross-validation\n",
    "- Create pipelines to perform grids search in less code\n",
    "- Improve upon a baseline model incrementally by adding in more complexity\n",
    "\n",
    "This lab will require using several Sklearn classes. It would be helpful to refer to appropriate documentation:\n",
    "- http://scikit-learn.org/stable/modules/preprocessing.html\n",
    "- http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler\n",
    "- http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html#sklearn.preprocessing.PolynomialFeatures\n",
    "- http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html\n",
    "- http://scikit-learn.org/stable/modules/pipeline.html#pipeline\n",
    "\n",
    "Also, here is a helpful tutorial that explains how to use much of the above:\n",
    "- https://civisanalytics.com/blog/data-science/2016/01/06/workflows-python-using-pipeline-gridsearchcv-for-compact-code/\n",
    "\n",
    "Like always, let's first load in the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['revenue', 'outcalls', 'incalls', 'months', 'eqpdays', 'webcap',\n",
       "       'marryyes', 'travel', 'pcown', 'creditcd', 'retcalls', 'churndep'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import KFold\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "cwd = os.getcwd()\n",
    "datadir = '/'.join(cwd.split('/')[0:-1]) + '/data/'\n",
    "\n",
    "\n",
    "\n",
    "data = pd.read_csv(datadir + 'Cell2Cell_data.csv', header=0, sep=',')\n",
    "\n",
    "#Randomly sort the data:\n",
    "data = data.sample(frac = 1)\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we're going to prep the data. From prior analysis (Churn Case Study) we learned that we can drop a few variables, as they are either highly redundant or don't carry a strong relationship with the outcome.\n",
    "\n",
    "After dropping, we're going to use the SkLearn KFold class to set up cross validation fold indexes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Prior analysis (from Churn Case study) has shown that we can drop a few redundant variables\n",
    "#We want to drop a few to speed up later calculations\n",
    "dropvar_list = ['incalls', 'creditcd', 'marryyes', 'travel', 'pcown']\n",
    "data_subset = data.drop(dropvar_list, 1)\n",
    "\n",
    "#Set up X and Y\n",
    "X = data_subset.drop('churndep', 1)\n",
    "Y = data_subset['churndep']\n",
    "\n",
    "#Use Kfold to create 4 folds\n",
    "kfolds = KFold(n_splits = 4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's use cross-validation to build a baseline model. We're going to use LR with no feature pre-processing. We're going to look at both L1 and L2 regularization with different weights. We can do this very succinctly with SkLearns GridSearchCV package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5851569126577355\n"
     ]
    }
   ],
   "source": [
    "#1st, set up a paramater grid\n",
    "param_grid_lr = {'C':[10**i for i in range(-3, 3)], 'penalty':['l1', 'l2']}\n",
    "\n",
    "#2nd, call the GridSearchCV class, use LogisticRegression and 'roc_auc' for scoring\n",
    "lr_grid_search = GridSearchCV(LogisticRegression(), param_grid_lr, cv = kfolds, scoring = 'roc_auc') \n",
    "lr_grid_search.fit(X, Y)\n",
    "\n",
    "#3rd, get the score of the best model and print it\n",
    "best_1 = lr_grid_search.best_score_\n",
    "print(best_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Next let's look at the best-estimator chosen to see what the parameters were\n",
    "lr_grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see if we can beat this by standardizing the features. We'll approach this using the GridSearchCV class but also build a pipeline. Later we'll extend the pipeline to allow for feature engineering as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5867346323935858\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "#Create a set of steps. All but the last step is a transformer (something that processes data). \n",
    "#Build a list of steps, where the first is StandardScaler and the second is LogisticRegression\n",
    "#The last step should be an estimator.\n",
    "\n",
    "steps = [('scaler', StandardScaler()),\n",
    "         ('lr', LogisticRegression())]\n",
    "\n",
    "#Now set up the pipeline\n",
    "pipeline = Pipeline(steps)\n",
    "\n",
    "#Now set up the parameter grid, paying close to the correct convention here\n",
    "parameters_scaler = dict(lr__C = [10**i for i in range(-3, 3)],\n",
    "                  lr__penalty = ['l1', 'l2'])\n",
    "\n",
    "#Now run another grid search\n",
    "lr_grid_search_scaler = GridSearchCV(pipeline, param_grid = parameters_scaler, cv = kfolds, scoring = 'roc_auc')\n",
    "lr_grid_search_scaler.fit(X, Y)\n",
    "\n",
    "\n",
    "#Again, print the score of the best model\n",
    "best_2 = lr_grid_search_scaler.best_score_\n",
    "print(best_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=0.01, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l1', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Let's see the model after scaling. Did the optimal parameters change?\n",
    "lr_grid_search_scaler.best_estimator_.steps[-1][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've built a pipeline estimator that performs feature scaling and then logistic regression, let's add to it a feature engineering step. We'll then again use GridSearchCV to find an optimal parameter configuration and see if we can beat our best score above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5918729713358607\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "#Create a set of steps. All but the last step is a transformer (something that processes data). \n",
    "# Step 1 - PolynomialFeatures\n",
    "# Step 2 - StandardScaler\n",
    "# Step 3 - LogisticRegression\n",
    "\n",
    "steps_poly = [('polyfeat', PolynomialFeatures()),\n",
    "         ('scaler', StandardScaler()),\n",
    "         ('lr', LogisticRegression())]\n",
    "\n",
    "#Now set up the pipeline\n",
    "pipeline_poly = Pipeline(steps_poly)\n",
    "\n",
    "#Now set up a new parameter grid, use the same paramaters used above for logistic regression, but add polynomial features up to degree 3. \n",
    "parameters_poly = dict(polyfeat__degree = [1, 2],\n",
    "                       polyfeat__interaction_only = [True, False],\n",
    "                       lr__C = [10**i for i in range(-3, 3)],\n",
    "                       lr__penalty = ['l1', 'l2'])\n",
    "\n",
    "#Now run another grid search\n",
    "lr_grid_search_poly = GridSearchCV(pipeline_poly, param_grid = parameters_poly, cv = kfolds, scoring = 'roc_auc')\n",
    "\n",
    "lr_grid_search_poly.fit(X, Y)\n",
    "best_3 = lr_grid_search_poly.best_score_\n",
    "print(best_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('polyfeat',\n",
       "  PolynomialFeatures(degree=2, include_bias=True, interaction_only=False)),\n",
       " ('scaler', StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
       " ('lr',\n",
       "  LogisticRegression(C=1, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l1', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False))]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Let's look at the best estimator, stepwise\n",
    "lr_grid_search_poly.best_estimator_.steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now make a bar chart to plot results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1191ce860>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAswAAAFtCAYAAAATeTzMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X90VOWB//HPwCQBEm1+GEBmskCY\nEDJxs8HMrEEwK4eNKSpTVIQIR9JNQ6qG1lIti7QE2V0xK1RXTEEDrYNbbaSlbuI5bSC7GKRUNw6V\nRavdpJhIMtIaEJdiFwLJ/f7ht7OOCc+kMiGA79c5nJPJfe69z8VceHO9d8ZmWZYlAAAAAP0aNtQT\nAAAAAC5kBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAA\nBgQzAAAAYGAf6gl82hVXXKEJEyYM9TQAAABwiWtvb9eRI0cijrvggnnChAkKBAJDPQ0AAABc4jwe\nz4DGcUsGAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYE\nMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABg\nQDADAAAABgMK5oaGBmVmZsrlcqmqqqrfMdu2bZPb7VZ2drYWLlwYtuz48eNyOp1aunTpuc8YAAAA\nOI/skQb09PSooqJCjY2Ncjqd8nq98vl8crvdoTGtra16+OGHtXfvXiUlJen9998P28aqVatUUFAQ\n/dkDAAAAgyziFebm5ma5XC6lp6crNjZWxcXFqqurCxuzefNmVVRUKCkpSZI0evTo0LJ9+/bp97//\nvW644YYoTx0AAAAYfBGDORgMKi0tLfTa6XQqGAyGjWlpaVFLS4umT5+u/Px8NTQ0SJJ6e3t13333\naf369VGeNgAAAHB+RLwlYyDOnDmj1tZWNTU1qbOzUwUFBXrjjTf0wx/+UDfeeKOcTqdx/ZqaGtXU\n1EiSurq6ojElAAAAICoiBrPD4VBHR0fodWdnpxwOR9gYp9Opa665RjExMZo4caImT56s1tZWvfLK\nK9qzZ482btyoEydOqLu7WwkJCX0eHCwvL1d5ebkkyePxROO4AAAAgKiIeEuG1+tVa2ur2tra1N3d\nrdraWvl8vrAxc+fOVVNTkyTpyJEjamlpUXp6up599lkdOnRI7e3tWr9+vRYvXnzWd9kAAAAALkQR\ng9lut6u6ulpFRUXKysrS/PnzlZ2drcrKStXX10uSioqKlJKSIrfbrZkzZ2rdunVKSUkZ9MkDAAAA\ng81mWZY11JP4JI/Ho0AgMNTTAAAAwCVuoN3JJ/0BAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMA\nAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQz\nAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBA\nMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAA\nBgQzAAAAYEAwAwAAAAYEMwAAAGAwoGBuaGhQZmamXC6Xqqqq+h2zbds2ud1uZWdna+HChZKk/fv3\na9q0acrOzlZOTo6ef/756M0cAAAAOA/skQb09PSooqJCjY2Ncjqd8nq98vl8crvdoTGtra16+OGH\ntXfvXiUlJen999+XJI0aNUrPPPOMMjIy9N577ykvL09FRUVKTEwcvCMCAAAAoijiFebm5ma5XC6l\np6crNjZWxcXFqqurCxuzefNmVVRUKCkpSZI0evRoSdLkyZOVkZEhSRo3bpxGjx6trq6uaB8DAAAA\nMGgiBnMwGFRaWlrotdPpVDAYDBvT0tKilpYWTZ8+Xfn5+WpoaOiznebmZnV3d2vSpElRmDYAAABw\nfkS8JWMgzpw5o9bWVjU1Namzs1MFBQV64403QrdeHD58WHfeeae2bt2qYcP6NnpNTY1qamokiSvQ\nAAAAuKBEvMLscDjU0dERet3Z2SmHwxE2xul0yufzKSYmRhMnTtTkyZPV2toqSTp+/LhuuukmPfTQ\nQ8rPz+93H+Xl5QoEAgoEAkpNTT2X4wEAAACiKmIwe71etba2qq2tTd3d3aqtrZXP5wsbM3fuXDU1\nNUmSjhw5opaWFqWnp6u7u1u33HKLFi9erHnz5g3KAQAAAACDKWIw2+12VVdXq6ioSFlZWZo/f76y\ns7NVWVmp+vp6SVJRUZFSUlLkdrs1c+ZMrVu3TikpKdq2bZtefvll+f1+5ebmKjc3V/v37x/0gwIA\nAACixWZZljXUk/gkj8ejQCAw1NMAAADAJW6g3ckn/QEAAAAGBDMAAABgQDADAAAABgQzAAAAYEAw\nAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAG\nBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAA\nYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMA\nAAAGBDMAAABgQDADAAAABgQzAAAAYDCgYG5oaFBmZqZcLpeqqqr6HbNt2za53W5lZ2dr4cKFoe9v\n3bpVGRkZysjI0NatW6MzawAAAOA8sUca0NPTo4qKCjU2NsrpdMrr9crn88ntdofGtLa26uGHH9be\nvXuVlJSk999/X5L0wQcfaM2aNQoEArLZbMrLy5PP51NSUtLgHREAAAAQRRGvMDc3N8vlcik9PV2x\nsbEqLi5WXV1d2JjNmzeroqIiFMKjR4+WJO3YsUOFhYVKTk5WUlKSCgsL1dDQMAiHAQAAAAyOiMEc\nDAaVlpYWeu10OhUMBsPGtLS0qKWlRdOnT1d+fn4oigeyLgAAAHAhi3hLxkCcOXNGra2tampqUmdn\npwoKCvTGG28MeP2amhrV1NRIkrq6uqIxJQAAACAqIl5hdjgc6ujoCL3u7OyUw+EIG+N0OuXz+RQT\nE6OJEydq8uTJam1tHdC6klReXq5AIKBAIKDU1NRzOR4AAAAgqiIGs9frVWtrq9ra2tTd3a3a2lr5\nfL6wMXPnzlVTU5Mk6ciRI2ppaVF6erqKioq0c+dOHTt2TMeOHdPOnTtVVFQ0KAcCAAAADIaIt2TY\n7XZVV1erqKhIPT09Ki0tVXZ2tiorK+XxeOTz+UJh7Ha7NXz4cK1bt04pKSmSpFWrVsnr9UqSKisr\nlZycPLhHBAAAAESRzbIsa6gn8Ukej0eBQGCopwEAAIBL3EC7k0/6AwAAAAwIZgAAAMCAYAYAAAAM\nCGYAAADAgGAGAAAADAhmAAAAwIBgBgAAAAwIZgAAAMCAYAYAAAAMCGYAAADAgGAGAAAADOxDPYEL\nxZoXf6233js+1NMAAAD43HGPu1yr52QP9TTOimAGAOBitnv3UM8gev7mb4Z6BkC/COb/70L+Vw0A\nAGd117VDPYPoec4a6hkA/eIeZgAAAMCAYAYAAAAMCGYAAADAgGAGAAAADAhmAAAAwIBgBgAAAAwI\nZgAAAMCAYAYAAAAMCGYAAADAgGAGAAAADPhobAAXJpttqGcQPRYf9wsAFzOuMAMAAAAGBDMAAABg\nwC0Z5xv/mxkAAOCiwhVmAAAAwIBgBgAAAAwIZgAAAMCAYAYAAAAMCGYAAADAgGAGAAAADAYUzA0N\nDcrMzJTL5VJVVVWf5X6/X6mpqcrNzVVubq62bNkSWrZ8+XJlZ2crKytLX//612XxVmQAAAC4iER8\nH+aenh5VVFSosbFRTqdTXq9XPp9Pbrc7bNyCBQtUXV0d9r1f/vKX2rt3rw4cOCBJmjFjhnbv3q3r\nr78+ekcAAAAADKKIV5ibm5vlcrmUnp6u2NhYFRcXq66ubkAbt9lsOnnypLq7u3Xq1CmdPn1aY8aM\nOedJAwAAAOdLxGAOBoNKS0sLvXY6nQoGg33Gbd++XTk5OZo3b546OjokSdOmTdPMmTN15ZVX6sor\nr1RRUZGysrKiOH0AAABgcEXlob85c+aovb1dBw4cUGFhoUpKSiRJv/3tb/X222+rs7NTwWBQu3bt\n0p49e/qsX1NTI4/HI4/Ho66urmhMCQAAAIiKiMHscDhCV4wlqbOzUw6HI2xMSkqK4uLiJEllZWXa\nt2+fJOmFF15Qfn6+EhISlJCQoNmzZ+uVV17ps4/y8nIFAgEFAgGlpqae0wEBAAAA0RQxmL1er1pb\nW9XW1qbu7m7V1tbK5/OFjTl8+HDo6/r6+tBtF3/xF3+h3bt368yZMzp9+rR2797NLRkAAAC4qER8\nlwy73a7q6moVFRWpp6dHpaWlys7OVmVlpTwej3w+nzZs2KD6+nrZ7XYlJyfL7/dLkubNm6ddu3bp\nL//yL2Wz2fTFL35Rc+bMGexjAgAAAKLGZl1gb4zs8XgUCASGehqDx2Yb6hlEz4X1o4NLDecKMDCc\nK8BnNtDu5JP+AAAAAAOCGQAAADAgmAEAAAADghkAAAAwIJgBAAAAA4IZAAAAMCCYAQAAAAOCGQAA\nADAgmAEAAAADghkAAAAwIJgBAAAAA4IZAAAAMCCYAQAAAAOCGQAAADAgmAEAAAADghkAAAAwIJgB\nAAAAA4IZAAAAMCCYAQAAAAOCGQAAADAgmAEAAAADghkAAAAwIJgBAAAAA4IZAAAAMCCYAQAAAAOC\nGQAAADAgmAEAAAADghkAAAAwIJgBAAAAA4IZAAAAMCCYAQAAAAOCGQAAADAgmAEAAAADghkAAAAw\nGFAwNzQ0KDMzUy6XS1VVVX2W+/1+paamKjc3V7m5udqyZUto2aFDh3TDDTcoKytLbrdb7e3tUZs8\nAAAAMNjskQb09PSooqJCjY2Ncjqd8nq98vl8crvdYeMWLFig6urqPusvXrxY3/72t1VYWKgTJ05o\n2DAuagMAAODiEbFem5ub5XK5lJ6ertjYWBUXF6uurm5AG3/rrbd05swZFRYWSpISEhI0atSoc5sx\nAAAAcB5FDOZgMKi0tLTQa6fTqWAw2Gfc9u3blZOTo3nz5qmjo0OS1NLSosTERN16662aOnWqvvWt\nb6mnpyeK0wcAAAAGV1Tuj5gzZ47a29t14MABFRYWqqSkRJJ05swZ7dmzR+vXr9drr72md955R36/\nv8/6NTU18ng88ng86urqisaUAAAAgKiIGMwOhyN0xViSOjs75XA4wsakpKQoLi5OklRWVqZ9+/ZJ\n+vhqdG5urtLT02W32zV37lz96le/6rOP8vJyBQIBBQIBpaamntMBAQAAANEUMZi9Xq9aW1vV1tam\n7u5u1dbWyufzhY05fPhw6Ov6+nplZWWF1v3www9DV4137drV52FBAAAA4EIW8V0y7Ha7qqurVVRU\npJ6eHpWWlio7O1uVlZXyeDzy+XzasGGD6uvrZbfblZycHLrtYvjw4Vq/fr1mzZoly7KUl5enJUuW\nDPYxAQAAAFFjsyzLGupJfJLH41EgEBjqaQwem22oZxA9F9aPDi41nCvAwHCuAJ/ZQLuTN0UGAAAA\nDAhmAAAAwIBgBgAAAAwIZgAAAMCAYAYAAAAMCGYAAADAgGAGAAAADAhmAAAAwIBgBgAAAAwIZgAA\nAMCAYAYAAAAMCGYAAADAgGAGAAAADAhmAAAAwIBgBgAAAAwIZgAAAMCAYAYAAAAMCGYAAADAgGAG\nAAAADAhmAAAAwIBgBgAAAAwIZgAAAMCAYAYAAAAMCGYAAADAgGAGAAAADAhmAAAAwIBgBgAAAAwI\nZgAAAMCAYAYAAAAMCGYAAADAgGAGAAAADAhmAAAAwIBgBgAAAAwIZgAAAMBgQMHc0NCgzMxMuVwu\nVVVV9Vnu9/uVmpqq3Nxc5ebmasuWLWHLjx8/LqfTqaVLl0Zn1gAAAMB5Yo80oKenRxUVFWpsbJTT\n6ZTX65XP55Pb7Q4bt2DBAlVXV/e7jVWrVqmgoCA6MwYAAADOo4hXmJubm+VyuZSenq7Y2FgVFxer\nrq5uwDvYt2+ffv/73+uGG244p4kCAAAAQyFiMAeDQaWlpYVeO51OBYPBPuO2b9+unJwczZs3Tx0d\nHZKk3t5e3XfffVq/fn0UpwwAAACcP1F56G/OnDlqb2/XgQMHVFhYqJKSEknSxo0bdeONN8rpdBrX\nr6mpkcfjkcfjUVdXVzSmBAAAAERFxHuYHQ5H6IqxJHV2dsrhcISNSUlJCX1dVlam5cuXS5JeeeUV\n7dmzRxs3btSJEyfU3d2thISEPg8OlpeXq7y8XJLk8Xg++9EAAAAAURYxmL1er1pbW9XW1iaHw6Ha\n2lo999xzYWMOHz6sK6+8UpJUX1+vrKwsSdKzzz4bGuP3+xUIBPp9lw0AAADgQhUxmO12u6qrq1VU\nVKSenh6VlpYqOztblZWV8ng88vl82rBhg+rr62W325WcnCy/338epg4AAAAMPptlWdZQT+KTPB6P\nAoHAUE9j8NhsQz2D6LmwfnRwqeFcAQaGcwX4zAbanXzSHwAAAGBAMAMAAAAGBDMAAABgQDADAAAA\nBgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAA\nAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDAD\nAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYE\nMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYDCgYG5oaFBmZqZcLpeqqqr6LPf7/UpNTVVu\nbq5yc3O1ZcsWSdL+/fs1bdo0ZWdnKycnR88//3x0Zw8AAAAMMnukAT09PaqoqFBjY6OcTqe8Xq98\nPp/cbnfYuAULFqi6ujrse6NGjdIzzzyjjIwMvffee8rLy1NRUZESExOjexQAAADAIIl4hbm5uVku\nl0vp6emKjY1VcXGx6urqBrTxyZMnKyMjQ5I0btw4jR49Wl1dXec2YwAAAOA8ihjMwWBQaWlpoddO\np1PBYLDPuO3btysnJ0fz5s1TR0dHn+XNzc3q7u7WpEmTznHKAAAAwPkTlYf+5syZo/b2dh04cECF\nhYUqKSkJW3748GHdeeedevrppzVsWN9d1tTUyOPxyOPxcAUaAAAAF5SIwexwOMKuGHd2dsrhcISN\nSUlJUVxcnCSprKxM+/btCy07fvy4brrpJj300EPKz8/vdx/l5eUKBAIKBAJKTU39TAcCAAAADIaI\nwez1etXa2qq2tjZ1d3ertrZWPp8vbMzhw4dDX9fX1ysrK0uS1N3drVtuuUWLFy/WvHnzojx1AAAA\nYPBFfJcMu92u6upqFRUVqaenR6WlpcrOzlZlZaU8Ho98Pp82bNig+vp62e12JScny+/3S5K2bdum\nl19+WUePHg19z+/3Kzc3dzCPCQAAAIgam2VZ1lBP4pM8Ho8CgcBQT2Pw2GxDPYPoubB+dHCp4VwB\nBoZzBfjMBtqdfNIfAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAw\nAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAG\nBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAA\nYEAwAwAAAAYEMwAAAGBAMAMAAAAGBDMAAABgQDADAAAABgQzAAAAYEAwAwAAAAYEMwAAAGBAMAMA\nAAAGAwrmhoYGZWZmyuVyqaqqqs9yv9+v1NRU5ebmKjc3V1u2bAkt27p1qzIyMpSRkaGtW7dGb+YA\nAADAeWCPNKCnp0cVFRVqbGyU0+mU1+uVz+eT2+0OG7dgwQJVV1eHfe+DDz7QmjVrFAgEZLPZlJeX\nJ5/Pp6SkpOgeBQAAADBIIl5hbm5ulsvlUnp6umJjY1VcXKy6uroBbXzHjh0qLCxUcnKykpKSVFhY\nqIaGhnOeNAAAAHC+RAzmYDCotLS00Gun06lgMNhn3Pbt25WTk6N58+apo6Pjz1oXAAAAuFBFvCVj\nIObMmaM77rhDcXFxeuqpp1RSUqJdu3YNeP2amhrV1NRIkn7zm9/I4/FEY1oXpry887Kbrq4upaam\nDu5OLuX/Thh6nCvAwJyHc+W8nCcS5wrOu/b29gGNixjMDocjdMVYkjo7O+VwOMLGpKSkhL4uKyvT\n8uXLQ+s2NTWFrXv99df32Ud5ebnKy8sHNGEMjMfjUSAQGOppABc8zhUgMs4TfN5FvCXD6/WqtbVV\nbW1t6u7uVm1trXw+X9iYw4cPh76ur69XVlaWJKmoqEg7d+7UsWPHdOzYMe3cuVNFRUVRPgQAAABg\n8ES8wmy321VdXa2ioiL19PSotLRU2dnZqqyslMfjkc/n04YNG1RfXy+73a7k5GT5/X5JUnJyslat\nWiWv1ytJqqysVHJy8qAeEAAAABBNNsuyrKGeBKKvpqaG21yAAeBcASLjPMHnHcEMAAAAGPDR2AAA\nAIABwQzgkvbggw/qqquuGuppAOfky1/+sm6++eahnkZEfr9fCQkJQz0NIOoI5ouU6Q/PCRMmyGaz\nyWazaeTIkZoyZYrWrVsn7r7Bhairq0v33HOPJkyYoLi4OI0ZM0azZs1SY2PjUE8toqamJtlsNh05\ncqTf5Q8++GDoXBw2bJjGjRunRYsWhb1VJz4fvvzlL4d+FmJiYpSenq77779fH3300Xmbwx//+Eet\nXLlSLpdLI0aM0BVXXKHp06frRz/60Xmbw7lob2+XzWY769vb+f3+0O+xzWbTmDFjNGfOHP36178+\nzzPFpSgqH1yCC09lZaXuvvtunTx5Uv/+7/+uu+++W5dffrm++tWvDvXUgDC33Xab/vjHP+r73/++\nXC6X3n//fe3evVtHjx4d6qmFdHd3KzY29jOtm5mZqaamJvX29urgwYOqqKjQ/Pnz9corr0R5lrjQ\n/e3f/q3+9V//VadPn9aePXtUVlamjz76SJs2bTov+7/rrru0d+9ePf7447rqqqt07Ngxvfrqq/rg\ngw/Oy/4H6lzOt1GjRungwYOyLEvBYFDLly/XTTfdpJaWls+8TUDiCvMl67LLLtPYsWM1YcIElZWV\nKScnRzt37hzqaQFhPvzwQ+3Zs0dVVVWaNWuWxo8fL6/Xq/vvv1/FxcWSPv7Lc+XKlRo/frzi4uKU\nnp6uDRs2SJJ6enr0la98RRMnTtTIkSOVkZGhRx55RL29vcb9Pv3003K73RoxYoQmT56sxx57LGwd\nm82m733ve7r11lsVHx+vlStXfuZjtNvtGjt2rMaNG6frrrtOS5Ys0auvvqrjx49/5m3i4hQXF6ex\nY8cqLS1NCxcu1KJFi/Rv//ZvkqSXX35Z11xzjUaMGKExY8Zo2bJl6u7u7nc7zzzzjFJSUnTq1Kmw\n7y9atKjP5yR8Un19vR544AHdfPPNmjBhgqZOnaq7775bFRUVoTGWZem73/2uMjIyFBcXJ6fTqQce\neCC0fMWKFcrMzNTIkSM1YcIELV++XCdPnjQe94svvqi8vDyNGDFCEydO1Le//e2wY5swYYIefPBB\nlZaWKjExUYsWLTJuz8Rms2ns2LG68sor5fF4tGzZMr377rv67//+78+8TUDiCvMlz7Is7d69W2+/\n/bYyMjKGejpAmISEBCUkJKi+vl4zZszQiBEj+owpKSnRnj179Pjjj2vq1Kl69913Q7c09Pb2yuFw\naNu2bUpNTVVzc7PKy8uVkpKir3zlK/3uc/PmzaqsrNQTTzyhvLw8vfnmm1qyZIliYmK0dOnS0Lg1\na9Zo7dq1Wr9+vWw2W1SO93e/+51++tOfavjw4Ro+fHhUtomL18iRI3X69GkFg0HNnj1bd955p/x+\nvw4ePKiysjINGzZM3/3ud/usd/vtt+vee+9VXV2d5s+fL0n6n//5H73wwgvG2yvGjh2rhoYG3X77\n7frCF77Q75iVK1dq06ZNevTRR1VQUKCuri69/vrroeXx8fH6wQ9+IIfDobfeekt33XWX4uLi9I//\n+I/9bm/Hjh1atGiRHn/8cRUUFOjQoUO66667dOrUKa1fvz407tFHH9V3vvMdBQKBqN0++OGHH+q5\n556TJMXExERlm/gcs3BRKikpsW666aZ+l40fP96KjY214uPjrZiYGEuSNWLECGvv3r3neZZAZD/5\nyU+spKQkKy4uzsrPz7fuu+8+69VXX7Usy7JaWlosSdbPf/7zAW/v7//+761Zs2aFXq9evdrKzs4O\nvU5LS7OeeeaZsHUee+wxKysrK/RakrV06dKI+3rppZcsSVZXV1e/y1evXm0NGzbMio+Pt0aOHGlJ\nsiRZX//61wd8PLg0fPrP7P/8z/+0UlJSrPnz51srV660XC6X1dPTE1r+9NNPW7GxsdZHH33U7/oV\nFRVWUVFR6PXGjRutMWPGWKdPnz7rHHbv3m05nU7LbrdbU6dOtSoqKqydO3eGlv/hD3+w4uLirE2b\nNg34uDZt2mRNmjQpbN7x8fGh19ddd531D//wD2HrvPDCC1Z8fLzV29trWdbHf2fdfPPNEffV1tZm\nSbJee+21fpc//fTTliQrPj7eGjVqVOh88/l8Az4e4Gy4JeMS9c1vflP79+/X7t27NXPmTK1evVrX\nXnvtUE8L6OO2227Te++9pxdffFGzZ8/WL3/5S+Xn52vt2rV6/fXXNWzYMM2cOfOs6z/55JPyeDxK\nTU1VQkKCHnvsMR06dKjfsV1dXero6NBXv/rV0NXthIQErVixQgcPHgwb6/F4wl5nZ2eHxs+ePXvA\nxzdp0iTt379fr732mh566CFdffXVWrt27YDXx6WjoaFBCQkJGjFihKZNm6aCggI98cQTevvtt5Wf\nn69hw/7vr+QZM2aou7tbv/28ShyeAAAGcUlEQVTtb/vd1pIlS9TY2KjOzk5J0g9+8AOVlJTIbrfr\n0KFDYT/ff/p5Kygo0DvvvKNdu3Zp/vz5amlp0Q033BB6tuWtt97SqVOnNGvWrLMew09+8hPNmDFD\nY8eOVUJCgpYtW3bW802S9u3bp4ceeihsPgsXLtRHH32k3/3ud6Fxnz7fZs+eHRqfnZ0d4Xf2/4wa\nNUr79+/Xvn379NRTTykjI0NPPfXUgNcHzoZbMi5RKSkpcrlccrlc2r59uzIyMnTNNdcYwwMYKiNG\njFBhYaEKCwtVWVmpsrIyPfjgg/rhD39oXO/555/XN77xDa1fv17XXnutLr/8cn3ve9/TCy+80O/4\nP92n/OSTT0b8B2R8fHzY65/97Gc6ffq0pI//V/pAxcbGyuVySfo4ultbW1VRUSG/3z/gbeDSUFBQ\noJqaGsXExGjcuHEDuk3gbLcD/dVf/ZWuvvpq+f1+zZ07V4FAIHS+jBs3Tvv37w+NTU5ODn0dExOj\n6667Ttddd51WrFihf/qnf9KqVavC7lM+m1dffVXFxcVavXq1HnvsMSUmJqq+vl7333//Wdfp7e3V\n6tWrdfvtt/dZlpqaGvr60+fbli1b9L//+7+hOQ+UzWYLnW9TpkzR4cOHdccdd+ill14a8DaA/hDM\nnwNJSUlaunSpli1bptdffz1q92MCg8XtduvMmTOaMmWKent79dJLL+mLX/xin3G/+MUvdM0114Td\ne/zpK8WfNGbMGI0bN04HDx7U4sWL/6w5jR8//s8afzbf+c53lJmZqa997WvKy8uLyjZxcRg1alQo\n5j4pKytL27ZtU29vb+gq8y9+8QvFxsZq0qRJZ93ekiVL9Mgjj+jIkSOaPn26MjMzJX38oGl/++mP\n2+2WJJ04cUJZWVmKi4vTf/zHf/T7zMvevXvlcDi0atWq0Pfeffdd4/avvvpq/eY3vxnwfP7E4XD8\nWePPZtmyZXr00Uf105/+VLfeemtUtonPJ4L5Inb8+PGwqwiSlJiY2O/Ye+65R//8z/+sH//4x6GH\nRIChdvToUd1+++0qLS1VTk6OLrvsMgUCAT3yyCOaNWuWcnJyNH/+fJWVlenxxx/X1Vdfrc7OTrW3\nt+vOO+/U5MmT5ff79fOf/1wul0u1tbXavXu3kpKSzrrPNWvW6Gtf+5oSExN144036vTp0/rVr36l\nYDA4oKts/XnzzTf7nHs5OTn9jp00aZK+9KUvadWqVfrZz372mfaHS8s999yjf/mXf9E999yje++9\nV++8845WrFihpUuXatSoUWdd74477tA3v/lNbdq0SU8++WTE/Vx//fW644475PF4lJKSorfeeksr\nV67UlClTlJWVpeHDh+vee+/VAw88oLi4OBUUFOjo0aPat2+f7r77bk2ePFnBYFDPPvuspk2bph07\ndkR8D+fKykrdfPPNGj9+vObPny+73a4333xTzc3NeuSRR/7s3ytJamlpkd0eni9Tpkzpd+zll1+u\nsrIyrV69WnPnzg277QX4swz1TdT4bEpKSkIPNHzy12233WaNHz/eWrduXZ91lixZYmVlZYU9WAIM\npZMnT1oPPPCA5fF4rMTERGvkyJGWy+Wyli1bZh09ejQ05lvf+pY1btw4KzY21kpPT7eeeOIJy7Is\n69SpU1ZpaamVmJhofeELX7BKS0utNWvWWOPHjw/t49MP/VmWZT333HPW1KlTrbi4OCsxMdGaPn26\n9aMf/Si0XJL14x//OOL8//TQX3+//vCHP/S7b8uyrL1791qSeBD3c8T0oLZlffxA3l//9V9bsbGx\n1ujRo61vfOMb1smTJyOu/3d/93fWZZddZp04cSLiHNauXWtNnz7dSklJseLi4qzx48dbZWVl1qFD\nh0Jjenp6rIcfftiaOHGiFRMTYzmdTmvlypWh5StWrLCuuOIKKz4+3rrlllusjRs3Wp9MiU8/9GdZ\nlrVjxw5rxowZ1siRI63LLrvMysvLC53DlmWd9e+sT/vTQ3/9/XrjjTf63bdlWda7775r2e1269ln\nn424D+BsbJbFx78BAHAxmj17tpxOpzZv3jzUUwEuadySAQDARebYsWPas2ePdu7cqf/6r/8a6ukA\nlzyCGQCAi8zUqVP1wQcfaO3atbrqqquGejrAJY9bMgAAAAADHhcFAAAADAhmAAAAwIBgBgAAAAwI\nZgAAAMCAYAYAAAAMCGYAAADA4P8By2mkWUhgEdUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "results = np.array([best_1, best_2, best_3])\n",
    "labs = ['LR', 'Scaler-LR', 'Poly-Scaler-LR']\n",
    "\n",
    "fig = plt.figure(facecolor = 'w', figsize = (12, 6))\n",
    "ax = plt.subplot(111)\n",
    "\n",
    "width = 0.25\n",
    "ind = np.arange(3)\n",
    "rec = ax.bar(ind + width, results, width, color='r')\n",
    "\n",
    "ax.set_xticks(ind + width)\n",
    "ax.set_xticklabels(labs, size = 14)\n",
    "ax.set_ylim([0.5, max(results)*1.1])\n",
    "\n",
    "plt.plot(np.arange(4), max(results) * np.ones(4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
